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ABSTRACT 


The  original  operational  test  orientation  of  this  research  is  noted  in  the  Background  paragraph 
of  Chapter  1 .  To  accomplish  the  original  objectives  it  was  necessary  to  acquire  considerable  data  on 
the  cost,  schedule  and  performance  success  of  programs  within  the  Engineering  and  Manufacturing 
Development  (EMD)  phase  of  development.  The  current  research  used  the  original  methodology 
and  expands  the  results  by  including  more  recent  programs  and  additional  parameters.  Hence  this 
report  contains  program  cost,  schedule,  and  performance  results  by  Service,  and  within  three  sepa¬ 
rate  year  groupings.  It  also  contains  a  comparison  between  the  Services’  Operational  Test  Activity 
(OTA)  test  report  and  the  Director  Test  and  Evaluation  (DOT&E)  independent  evaluation  of  the 
Service  test  report. 


IX 


CHAPTER  1 
INTRODUCTION 


Background 

In  1994,  the  Office  of  the  Director,  Operational  Test  and  Evaluation  (DOT&E),  asked  the  Test 
and  Evaluation  Department  (T&E  Department)  to  research  the  relationship,  if  any,  between  the 
number  of  test  articles  used  in  the  Engineering  and  Manufacturing  Development  (EMD)  Phase  of  a 
program  acquisition,  and  the  success  of  that  program  in  EMD.  For  major  defense  acquisition  pro¬ 
grams  which  include  a  Low  Rate  Initial  Production  (LRIP)  effort  within  the  EMD  phase,  the  DOT«&E 
is  responsible  for  approving  the  number  of  test  articles  required  for  the  Initial  Operational  Test  and 
Evaluation  (lOT&E)*  accomplished  near  the  end  of  EMD.  Since  1991,  Congressional  Law  requires 
the  DOT&E  to  specify  at  Milestone  11,  the  number  of  test  articles  required  for  this  test.^  The  deter¬ 
mination  of  test  quantities  is  a  difficult  trade-off  among  several  factors;  the  office  of  the  DOT&E 
wanted  to  know  if  data  was  available  that  would  help  in  making  this  important  decision.  Indications 
were  that  factual  references  or  metrics  relating  to  the  subject  were  nonexistent.^ 

This  research  was  completed  and  the  results  published  in  a  Defense  Systems  Management 
College  Technical  Report  dated  May  1995,'*  hereafter  called  the  original  research  report.  The  inter¬ 
est  in  the  original  research  methodology  and  conclusions  have  resulted  in  a  follow-on  research  effort 
described  in  this  technical  report. 

Overview 

This  research  extends  the  prior  research  in  two  important  ways.  In  order  to  answer  the  specific 
T&E  question  it  was  necessary  to  devise  a  research  methodology  and  accumulate  considerable 
EMD  program  management  data.  This  larger,  general  management  data,  including  comparative 
Operational  Test  Activity  (OTA)^/DOT&E  operational  test  evaluations,  has  become  the  focus  of 
current  research.  The  initial  research  sought  to  identify  the  test  article  relationship  to  EMD  program 
success  in  terms  of  cost  and  schedule  overruns  only.  During  dissemination  of  the  results  of  the 
original  research,  the  question  “What  about  performance?”  was  asked.  We  originally  established 
criteria  for  cost,  schedule,  and  performance  success,  but  collected  no  performance  data.  The  current 
research  adds  performance  data  to  the  database;  this  is  the  second  significant  addition  to  the  re¬ 
search  methodology. 

In  most  other  respects  this  research  uses  the  original  research  methodology.  The  original  data¬ 
base  for  24  programs  that  completed  the  EMD  phase  has  been  extended  to  53  programs  with  41 
programs  having  nearly  complete  data  within  the  database.  Programs  continue  to  be  added  as  they 
near  completion  of  their  EMD  phase. 

All  data  used  within  this  report  are  unclassified.  The  numbers  of  data  points  vary  between 
parameters  due  to  the  non-availability  of  some  data. 

It  is  important  to  recognize  that  our  research  approach  is  a  measure  of  EMD  program  manage¬ 
ment  success  in  terms  of  cost,  schedule  and  performance  metrics  rather  than  the  eventual  weapon 


1-1 


system  success.  There  was  no  attempt  to  survey  the  effectiveness  of  the  systems  in  their  operational 
roles  in  the  field. 

The  subsequent  chapters  of  this  report  are  organized  as  follows.  Portions  of  the  original  re¬ 
search  report  that  remain  relevant  are  repeated.  Chapter  2  discusses  recent  literature  search  efforts 
and  repeats  some  of  those  from  the  original  report.  Chapter  3  essentially  repeats  the  description  of 
the  original  research  methodology  and  an  overview  of  the  spreadsheet  contained  in  Appendix  C. 
Chapter  4  describes  the  hmitations  and  assumptions  implicit  in  this  research.  Chapter  5  is  an  analysis 
of  the  results  of  this  effort.  Chapter  6  is  a  discussion  of  opinions  resulting  from  our  viewpoint  of  the 
facts  contained  in  the  data  bank. 

Figure  1  is  intended  to  show  the  relationship  of  the  EMD  phase,  between  Milestone  II  and 
Milestone  III,  to  the  total  system  acquisition  and  test  and  evaluation  activities  therein. 
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DETERMINATION  |  PHASED  PHASE  I  PHASE  II  PHASE  III 

OF  MISSION  I  CONCEPT  PROGRAM  ENGINEERING  &  PRODUCTION, 

NEED  I  EXPLORATION  DEFINITION  &  MANUFACTURING  FIELDINGIDEPLOYMENT 

I  RISK  REDUCTION  DEVELOPMENT  AND  OPERATIONAL  SUPPORT 


FIGURE  1.  Test  and  Evaluation  in  the  Acquisition  Process 
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CHAPTER! 

REVIEW  OF  RELATED  LITERATURE 


Discussion 

Most  of  the  literature  search  conducted  for  this  continuing  research  effort  was  accomplished 
in  the  original  phase.  The  reader  is  referred  to  the  original  research  report  for  details.  Newer 
literature  research  is  listed  here  together  with  prior  research  that  remains  directly  relevant  to 
current  efforts.  The  Department  of  Defense  (DoD)  Directives,  particularly  those  associated  with 
the  Selected  Acquisition  Report  (SAR),®  operational  test  reports,  and  other  Government  guid¬ 
ance  documents  were  utilized.  Some  of  the  prior  reports  most  relevant  and  useful  are  discussed 
below. 

Related  References 

Rich  and  Dews’  Improving  the  Military  Acquisition  Process,''  contains  a  finding  that  is  dif¬ 
ferent  than  our  own.  It  states  for  programs  in  the  1960s,  cost  overruns  approximated  45  percent, 
schedule  overruns  15  percent,  and  performance  shortfalls  5  percent.  “These  findings  support  the 
conventional  wisdom  that  when  acquisition  problems  arise  cost  is  the  constant  most  easily  relaxed 
and  schedule  is  next,  whereas,  performance  goals  are  adhered  to  most  closely.”  Our  own  findings, 
detailed  later,  show  average  cost  overrun  of  40  percent,  average  schedule  overrun  of  62  percent, 
and  an  average  performance  success  rating  of  3.8  (76  percent)  out  of  a  possible  5.0  (100  percent) 
points.  A  possible  explanation  for  this  variance  is  the  time  period  being  used;  1960s  by  Rich  and 
Dews,  and  the  1980s-1996  (mostly  1988-1996),  used  by  us.  In  the  former  time  period,  the  Cold 
War  was  raging,  and  each  new  weapon  system  was  needed  by  the  war  fighter  as  soon  as  possible. 
In  the  later  period,  the  Cold  War  was  “won”  and  schedule  became  relatively  less  important  than 
cost.  In  our  studies  performance  has  always  had  a  higher  success  rating  than  cost  or  schedule,  but 
our  criteria  system  probably  is  different  than  Rich  and  Dews. 

Drezner,  et  al.,®  in  a  1993  study  of  128  weapon  system  programs  three  or  more  years  into  the 
Engineering  and  Manufacturing  Development  (EMD)  phase  concluded,  “The  weighted  average 
cost  growth  of  Development  Estimate  baseline  programs,  (Milestone  (MS)  II). .  .as  of  December 
1990  is  20  percent.”  Our  own  study  started  at  the  same  point  but  continued  through  all  of  EMD 
concluding  with  MS  III  actual  cost  data.  When  you  consider  that  the  average  duration  of  the 
EMD  phase  for  major  programs  is  7.4  years,^  and  a  large  degree  of  the  cost  overrun  presumably 
occurs  toward  the  end  of  EMD,  Drezner ’s  figure  is  close  to  our  own  higher  figure,  for  the  greater 
length  of  time. 

Other  studies  comment  on  requirements  generation  and  the  program  risk  evaluation  process, 
indicating  both  should  be  reviewed.  Wiles'®  indicates  the  requirements  generation  and  management 
process  led  to  unrealistic  operational  requirements.  Earlier,  David  Packard"  stated  all  too  often, 
requirements  for  new  weapon  systems  have  been  overstated.  Commenting  on  the  Advanced  Me¬ 
dium  Range  Air  to  Air  Missile  (AMRAAM)  program’s  spotted  history,  Mayer’s'^  comments  “. .  .the 
chief  cause  of  AMRAAM’s  woes  is  that  managers  vastly  over-sold  the  program  in  terms  of  cost  and 
schedule.” 
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Approximately  600  titles  were  extracted  from  the  Defense  Technical  Information  Center 
(DTIC)  and  other  Acker  Library  research  reference  databases.  An  estimated  100  bibliographies 
were  ordered,  and  approximately  105  reports  were  scanned  or  read  in  support  of  this  research 
effort.  With  the  exception  of  the  dissimilar  findings  discussed  previously,  the  great  majority  of 
prior  research  findings  were  either  closely  supportive,  or  generally  supportive,  of  the  findings  of 

this  research. 

More  recently,  two  articles  have  been  prepared  for  publication  based  on  this  research.  The 
first,  “Management  and  Measurement”  appeared  in  the  May- June  1999  issue  of  Program  Man¬ 
ager  magazine.  It  suggested  two  major  changes  to  the  DoD  acquisition  system  occurred  in  199 
which  considerably  improved  the  operational  test  performance  of  systems  in  EMD.  The  second 
article  “Baselining  Acquisition  Reform  (AR),”  is  approved  for  publication  in  an  upcoming 
sition  Review  Quarterly.  It  suggests  a  date  when  AR  first  became  effective  in  the  field,  an  t  e 
cost  schedule  and  performance  success  levels  at  that  time.  At  some  future  date  using  this  researc 
methodology,  another  measurement  can  be  taken  to  record  the  degree  of  acquisition  improvement 
that  can  be  attributed  to  AR  efforts. 
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CHAPTERS 

METHODOLOGY 


Concept  and  Approach 

As  an  underlying  concept  to  this  study,  we  used  only  official  data  available  to  DoD  principals 
and  staff  at  the  Defense  Acquisition  Board  (DAB)  Milestone  (MS)  II  and  III  meetings.  The  ratio¬ 
nale  was  that  acquisition  decisions  were  based  largely  on  this  data,  whether  it  was  complete  or 
incomplete,  clear  or  hazy,  right  or  wrong,  with  regard  to  estimates  stated.  We  therefore  limited 
our  data  sources  to  the  program  SARs  for  the  years  the  program  was  in  EMD,  and  the  Initial 
Operational  Test  and  Evaluation  (lOT&E)  reports  of  the  Service  OTAs  and  the  Beyond  Low  Rate 
Initial  production  (BLRIP)  independent  evaluations  issued  by  the  DOT&E. 

The  current  research  is  a  follow-on  effort  to  the  original  effort.  So  that  newer  data  would  be 
compatible  with  the  original  data,  we  felt  the  original  criteria  for  cost,  schedule  and  performance 
success  must  be  maintained.  Experience  has  shown  us  no  problems  have  occurred  using  this 
approach. 

For  purposes  of  this  research  study,  the  success  of  a  DoD  acquisition  program  will  be  cat¬ 
egorized  in  one  of  five  ways.  Either  the  program  is  Very  Successful,  Successful,  Fairly  Successful, 
Marginally  Successful,  or  Not  Successful.  The  subjective  description  of  these  categories  follows. 
For  the  original  report,  only  the  portion  of  the  definition  relating  to  cost  or  schedule  overruns 
apply.  The  remainder  of  the  definitions  apply  to  the  current  follow-on  effort. 

Very  Successful  (Score  of  5) 

There  are  few,  if  any,  system  shortcomings.  The  MS  II  program  budget  and  program  schedules 
were  essentially  adhered  to. 

The  DOT&E  MS  III  BLRIP  report  was  positive.  The  Service  lOT&E/OPEVAL  report  was 
positive,  namely  effective  and  suitable  without  caveat.  (If  not  suitable,  the  deficiencies  could  be 
corrected  without  major  impact — i.e.,  no  SAR  breach.) 

Successful  (Score  of  4) 

The  ADM  from  MS  II  and  MS  IB  DABs  was  straightforward.  There  were  system  shortcom¬ 
ings.  The  MS  II  program  budget  and  schedule  were  slipped,  but  not  by  more  than  30  percent  in  cost 
and  12  months  in  schedule. 

The  DOT&E  MS  III  BLRIP  report  was  positive.  The  Service  lOT&E/OPEVAL  (Operational 
Evaluation)  report  was  positive.  The  overall  evaluation  was  effective  and  suitable,  with  perhaps  a 
few  marginally  suitable  parameters. 
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Fairly  Successful  (Score  of  3) 

The  ADM  from  the  MS  n  and  MS  HI  DABs  contained  FoWem  .^.viSldSr 
shortcomings  were  listed;  a  few  could  be  critical.  The  MS  H  program  budget  and  schedule  M  to  be 
revised,  but  were  within  45  percent  of  the  MS  Hbudget  and  no  mote  than  18  months  behind  the  MS 

n  schedule. 

The  DOT&E  MS  ffl  BLRIP  report  contained  a  few  negative  comments.  The  Service  lOT&E/ 
OPEVAL  report  could  be  marginally  effective  and  marginally  suitable. 

Maryinallv  Successful  (Score  of  2) 

The  ADM  from  the  MS  H  and  MS  IE  DABs  indicated  major  performance  and  suitabilky  prob¬ 
lems  existed  The  program  probably  would  be  canceled  on  the  basis  of  perform^ce  o  a  e, 
other  L  hing  considered.  The  MS  n  program  budget  and  scWule  was  revved 

more  than  once,  and  is  now  up  to  60  percent  overrun  m  cost  and  two  years  behind  the  ong 

schedule. 

The  exit  criteria  of  the  MS  U  ADM  were  not  completely  met.  An  outcome  of  the  MS  HI  DAB 
would  be  to  delay  entry  into  full  rate  production. 

The  DOT&E  MS  IB  BLRIP  report  was  marginally  effective  and/or  marginally  suitable.  The 
Service  lOT&E/OPEVAL  recommended,  at  best,  the  system  was  potentially  effective  and  poten¬ 
tially  suitable. 

Nnt  Successful  (Score  of  1) 

The  ADM  from  the  MS  H  DAB  reluctantly  approved  the  continuation  of  the  program  into 
EMD^t  held  .he  program  in  d,e  Demonshah"^  P'-f, 

percent  overrun,  and  the  program  is  more  than  2  years  behind  schedule.  A  DOT&E  BLMP  r  p 
will  say  it  is  not  effective  and  not  suitable.  This  category  would  also  include  programs  that  have  in 

fact  been  terminated. 

For  programs  that  have  not  had  their  MS  IB  DAB  review  as  yet,  their  success  will  be  judged  on 
the  general  approach  discussed  herein,  and  on  the  available  official  documentation. 

Cost  and  Schedule  criteria  are  objective;  they  use,  in  general,  the  standard  DoD  decrements  for 
SAR  reporting:  15  percent  in  cost  growth  and  6  months  in  schedule  slippage, 
tion  is  subjective,  but  controlled.  Originally  all  performance  success  ratings  were  assigned  by 
researcher  Most  recently,  the  performance  success  rating  is  to  be  assigned  by  a  three-member  Sub 
ject-Matter  Expert  (SME)  Panel  of  T&E  Department  faculty  members  in  accordance  with  estab 
lished  rating  criteria  and  initial  standardization  procedures. 

A  standardized  approach  to  present  the  data  has  been  established.  The  major  division  in  pre- 
senting"is  general  managLent  metrics  followed  by  operational  test  metrics.  Examples  of 
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the  former  include  cost  and  schedule  overruns  and  of  the  latter,  comparisons  between  OTA  and 
DOT«&E  evaluations.  Next  within  each  category  we  list  the  overall  metric  for  the  years  covered 
(1980-1996).  Then  the  first  sub-set  of  the  metric  is  by  year  group  (1980-1988,  1989-1992,  1993- 
1996),  and  then  by  individual  Service. 


In  reviewing  this  data,  particularly  bar  graphs,  the  reader  is  reminded  that  overrun  data  is 
the  inverse  of  success  data.  For  example,  a  bar  graph  showing  a  large  overrun  would  be  relatively 
tall.  The  associated  success  rating  for  this  cost  or  schedule  parameter  would  be  relatively  short. 

Spreadsheet  Description 


The  database  is  maintained  on  an  Excel  spreadsheet  with  columns  representing  parameters 
recorded,  and  rows  representing  the  number  of  programs  entered  to  date.  Currently  about  8-12  new 
programs  are  added  each  year.  The  first  12  columns  are  called  primary  data,  that  is,  the  identification 
of  programs  and  their  EMD  cost,  schedule,  and  performance  results.  The  remaining  columns  are 
expanded  data,  required  to  arrive  at  the  program  success  ratings. 


Taxonomy  of  weapon  system  types  was  created  for  use  within  the  study  and  consists  of: 


A/C: 

Veh: 

Elec-EW-A: 

Elec-CNR-A: 

Elec-CNR-G: 

Mis/Mu-A: 

Mis/Mu-G: 


Aircraft 

Vehicle 

Electronic-Electronic  Warfare-Airborne 
Electronic-Comm/Nav/Radar-Airbome 
Electronic-Comm/Nav/Radar-Ground 
Missile/Munition- Airborne 
Missile/Munition-Ground 


Consistent  with  the  idea  that  all  data  should  be  that  which  are  available  to  OSD-level  decision 
makers,  cost  data  for  all  systems  were  collected  from  the  system  SARs.  As  the  period  of  interest  was 
EMD,  the  RDT&E  and  Procurement  costs  were  collected  for  the  period  from  MS  II  through  MS 
III.  Planned  costs  were  those  reported  in  the  SAR  following  MS  II;  actual  costs  were  those  reported 
in  the  SAR  following  MS  III. 

In  theory,  one  could  go  to  the  program  SARs  and  retrieve,  rather  directly,  the  planned  and 
actual  cost  of  EMD.  In  practice,  the  retrieval  of  costs  from  the  SARs  is  not  simple.  For  one  thing, 
costs  are  reported  in  several  places  in  the  SAR:  for  example,  sections  11,  13,  14  and  16.  For 
another,  costs  are  reported  in  both  Base  Year  and  in  Then  Year  dollars;  the  inflators  for  each 
appropriation  are  not  always  explicit.  And  yet  another,  the  costs  are  reported  by  fiscal  year;  EMD 
need  not  start  or  end  in  conjunction  with  fiscal  years.  This  leaves  the  analyst  with  the  problem  of 
determining  how  much  of  a  particular  fiscal  year’s  costs  should  be  allocated  to  EMD  for  the 
database. 


Base  Year  vs.  Then  Year.  In  principle,  the  use  of  Base  Year  figures  will  remove  the  effect  of 
inflation  on  the  figures,  making  it  easier  to  perform  analyses.  In  the  case  of  the  EMD  database,  the 
EMD  costs  are  used  as  a  ratio  of  planned  to  actual.  For  most  programs,  the  same  inflators  will  be 
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used  in  both  sets  of  figures,  thereby  yielding  the  same  results  from  either  Base  Yej 
figures.  For  those  programs  in  which  actual  EMD  fiscal  years  were  significantly  longer  th^  planned, 
when  inflation  might  have  had  an  effect  on  the  ratios,  use  of  either  figure  will  still  identify  a  signi 
cant  overrun.  In  addition,  several  programs  changed  their  Base  Year  for  Si^-reporting  during 
EMD.  This  creates  additional  difficulties  in  using  Base  Year  figures  for  the  database.  For  all  the 
above  reasons,  this  analysis  used  Then  Year  costs. 

SAR  Section.  Sections  J 1, 13  and  14  do  not  separate  costs  by  fiscal  year;  it  is  "Ol  “ 

determine  costs  for  EMD  alone  from  these  figures.  Section  16  is  the  sole  portron  of  the  S  AR 
records  costs  by  fiscal  year.  Start  of  EMD  (MS  ID,  found  in  section  9.  determrriM  the 
year  to  use.  It  MS  II  is  in  the  middle  of  a  fiscal  year,  costs  can  be  prorated  ^ 

determines  the  end  of  EMD.  If  MS  HI  is  in  the  middle  of  a  fiscal  year,  the  end  of  lOT&E  can  be  used 

fo  determine  the  end  of  EMD  expenditures.  With  the  above  adjustments,  sectron  16  figures  were 
used  for  the  SAR  cost  data. 

RDT&E  Costs.  For  some  programs  it  is  difficult  to  determine  what  RDT&E  costs  during  the 
period  of  EMD  actually  were  incurred  in  development  of  the  program.  In  particu  ar,  w  en  e 
program  includes  several  models,  or  includes  preplanned  product  improvements,  the  program  can 
incur  RDT&E  costs  supporting  these  others  projects  during  the  period  that  the  basic 
is  in  EMD.  An  excellent  example  of  this  is  the  Joint  Tactical  Information  Distribution  System 
(JTIDS)  program,  in  which  class  2H/2M  development  was  initiated  before  class  2  completed  MS 
m  SAR^data  can  not  isolate  the  costs  of  the  basic  program,  except  m  those  cases  where  the 
progrl  office  identified  the  parts  of  the  project  separately  in  the  SAR.  Addition^,  for  pro^ 
graL  with  preplanned  product  improvements,  total  cost  figures  (such  as  are  m  sections  1 1  13  or 
fZmZ  deL  the  basic  development.  In  those  instances  where  cost  definition  was  not  clear, 
best  estimates  of  allocation  to  the  basic  development,  year  by  year,  were  made. 

Prnrnrp.ment  Costs.  The  SARs  do  not  Separately  identify  the  cost  of  LRIP  productmn.  The  best 
approximation  is  to  use  all  procurement  costs  for  those  years  before  MS  HI;  this  is  predicated  on 
legal  constraint  against  using  full-rate  production  funds  before  that  decision.  Procurement  costs  also 
hfve  the  same  separation  problem  as  discussed  for  RDT&E  above,  for  programs  with  several  mod¬ 
els;  the  same  solution  was  used. 

pnTX.B  V,  Prfwiimment.  There  remains  a  significant  problem  in  detemuning  the  costs  to  be 
used  in  calculating  EMD  Cost  Overrun.  For  any  program  with  more  than  a  ‘tetm,  e 

Procurement  costs  in  EMD  will  be  significantly  (in  some  cases  overwhelmingly)  I”*" 
costs.  This  results  in  the  program  being  scored  for  EMD  Cost  Over™ 

mentcosts.  The  problem  is  thatalaigenumberoILRIPitemsmayrndtcateavery  bad  prog;™(they 

need  to  continue  LRIP  to  keep  the  line  warm  while  the  program  keeps  trying  » 

example).  A  large  number  of  LRIP  items  may  aUo  indicate  a  very  good  progr™  (testmg  ■" 

been  !o  successful  the  services  demand  the  item  in  the  field  before  MS  m  or  “atnpte)  Jhus. 

overwhelming  RDT&E  with  Procurement  EMD  Costs  produces  equtv^a^  data.  For 

EMD  costs  for  RDT&E  and  Procurement  were  entered  separately  in  the  database,  with  RDT&E 

costs  alone  being  used  for  the  EMD  Cost  Overrun  calculation.  Analysis  of  Procurement  costs  may 

still  be  performed,  if  desired,  since  the  data  are  present  in  the  database.  It  should  be  noted  that  a 
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sensitivity  analysis  was  performed  on  the  data  from  the  systems  reported  in  1995.  The  result  was 
that  use  of  RDT&E  costs  alone  produced  the  same  EMD  Cost  Success  factor,  within  a  score  1  more 
or  less,  in  13  out  of  14  programs  (see  Appendix  B). 
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CHAPTER  4 

LIMITATIONS  OF  STUDY  AND  ASSUMPTIONS 


Limitations 

By  definition,  this  study  concerns  itself  with  the  Engineering  and  Manufacturing  Development 
(EMD)  phase  exclusively,  since  that  is  where  LRIP  resides. 

The  data  for  this  study  were  restricted  to  that  contained  within  the  Selected  Acquisition  Report 
(SAR)  for  the  years  in  which  the  program  was  in  EMD,  and  the  lOT&E  and  BLRIP  operational  test 
reports.  This  was  an  early,  deliberate  design  decision.  The  rationale  was  we  wanted  to  use  only  the 
data  available  to  the  decision  makers  at  the  time  the  decision  was  made. 

In  some  instances,  the  data  within  the  SARs  were  incomplete  and  in  some  cases  the  type  of  data 
contained  within  the  SARs  varied  between  programs.  This  is  not  considered  unusual  given  the  elapsed 
time  since  some  of  the  programs  had  their  DAB  meetings  and  the  dynamics  of  the  process. 

SAR  reports  are  required  for  Acquisition  Category  I  (ACAT  I)  programs  only,  whereas  the 
programs  within  the  database  are  for  all  programs  that  have  completed  EMD  and  are  on  the  DOT&E 
Oversight  List.  Therefore  it  was  not  possible  to  obtain  cost  and  schedule  data  for  some  programs. 

Even  for  programs  with  SARs  it  was  not  always  easy,  or  even  possible,  to  extract  cost  data 
for  the  program  which  was  initiated  at  MS  II.  Several  programs  were  changed  significantly  during 
EMD,  to  the  point  that  the  cost  figures  for  the  initial  program  were  lost  in  the  actual  reported 
costs.  One  example  is  the  JTIDS,  which  shows  the  difficulties  in  abstracting  cost  data  from  the 
SARs  of  programs  which  have  undergone  significant  restructuring.  As  initiated  at  MS  II  in  1 98 1 , 
this  was  a  joint  Army/Air  Force  demonstration  program  for  class  2  terminals  only,  with  a  follow- 
on  if  the  demonstration  was  successful.  The  Navy  had  its  own  program.  In  1985  the  Navy  was 
directed  to  use  the  Air  Force  terminals.  Also  in  1985,  the  Army  redirected  its  program  to  develop 
2M  (data  only)  terminals.  In  1986  the  Air  Force  added  2H  terminals,  initially  for  the  E-3A  only, 
with  other  platforms  to  be  added  later.  One  result  of  all  this  restructuring  was  that  MS  IIIA  in 
1989  was  only  for  Navy/ Air  Force  class  2  terminals.  Throughout  EMD  the  SAR  cost  data  were 
revised  and  reordered  to  reflect  these  programmatic  decisions.  Comparing  planned  costs  for  EMD 
to  actual  costs,  as  well  as  comparing  planned  and  actual  schedule  information,  using  only  the 
SAR,  was  virtually  impossible  under  these  circumstances.  In  some  cases  a  best-effort  set  of  fig¬ 
ures  was  derived;  in  others,  no  reasonably  comparable  set  of  figures  was  possible  and  no  data 
were  entered. 

The  degree  to  which  software  must  he  developed  for  a  system  is  another  important  variable, 
which  would  affect  program  success.  We  had  planned  to  include  this  as  a  variable,  but  discovered 
that  the  SARs  contained  very  little  data  in  this  regard. 

Program  concurrency — ^the  overlapping  of  program  development  and  program  production 
phases — is  a  concern.  This  study  generated  little  data  relevant  to  concurrency;  although  again,  pro¬ 
gram  concurrency  intuitively  would  have  a  major  impact  on  program  EMD  success. 


4-1 


The  current  research  may  be  constrained  by  the  decision  to  use  the  original  research  methodol¬ 
ogy.  The  reason  for  this  decision  has  been  explained  earlier. 

The  basic  intent  of  this  research  is  to  estimate  the  degree  of  accuracy  by  program  managers 
between  their  planned  and  their  actual  cost,  schedule,  and  performance  figures.  Extern^  influences, 
not  necessarily  under  the  control  of  the  program  manager,  would  confound  this  analysis.  Probably 
some  degree  of  external  influence,  usually  attributed  to  Congress,  does  exist.  J.  Ronald  Fox  in  his 
1988'^  book  listed  other  potential  external  inputs,  such  as:  “. .  .the  DoD  often  sets  optimistically  low 
cost  estimates  for  proposed  programs. . ..  Only  much  later  does  it  become  clear  that  these  programs 
will  require  more  money  than  originally  estimated”’'^  or  “...  “cut  insurance”  is  another  budgetary 
technique.  In  this  case... DoD  inflates  the  cost  of  selected  programs  to  ensure  that  even  after  ex¬ 
pected  congressional  cuts,  desired  funding  levels  will  remain  intact,”  and  ...“There  is  simply  no 
need  to  lie;  one  only  has  to  be  optimistic.”'^  Any  degree  of  situational  estimating  would  have  an 
impact  on  pure  estimating  accuracy. 

Recently  there  have  been  major  changes,  improvements,  etc.  to  the  “5000  Series  Acquisition 
Regulations”  which  regulate  the  mainstream  DoD  weapon  systems  development  this  research  cov¬ 
ers.  This  includes  the  large  number  of  major  improvements  commonly  called  Acquisition  Reform 
(AR)  and  pilot  programs  such  as  Advanced  Concept  Technology  Demonstrator  (ACTD).  The  effect 
and  relation  of  these  new  initiatives  on  our  research  is  not  discussed. 


Assumptions 

In  the  established  DoD  acquisition  system  for  ACAT I  programs,  the  only  oversight  documents 
required  on  a  routine  basis  by  the  Office  of  the  Secretary  of  Defense  (OSD)  are  the  SARs  and 
DOT&E  BLRIP  evaluation. 

The  validity  of  this  study  depended  greatly  on  how  important  the  data  contained  within  the 
SARs  was  to  the  actual  decisions  made  at  DAB  meetings.  It  was  assumed  the  information  was 
central  and  vital  to  the  decision  process. 


This  study  is  the  only  known  research  that  attempts  to  combine  and  evaluate  EMD  cost  and 
schedule  together  with  performance  as  measured  by  OTA  and  DOT&E  operational  tests  during 
EMD. 

Also  this  study  attempts  to  identify  program  EMD  costs  only.  The  S  AR,  using  development 
and  production  estimate  baselines,  contain  cost  to  date,  and  cost  to  complete  for  the  entire  program. 
It  is  assumed  you  cannot  easily  extract  EMD  costs  only  from  the  SARs. 
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CHAPTERS 

FINDINGS 


Primary  Findings 

Table  1  lists  the  primary  findings  of  this  study.  Most  findings  are  discussed  in  greater  detail  later 
in  this  Chapter,  and  in  Chapter  6.  Also,  several  findings  are  illustrated  in  the  figures  used  herein. 
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General  Management  Metrics 

•  The  average  cost  and  schedule  overruns  for  33  program  whose  EMD  phase  ended 
between  1980  and  1996  was  40  percent  in  cost  and  62  percent  in  schedule. 

•  The  degree  of  cost  and  schedule  overruns  can  be  directly  related  to  the  Cold  War 
years,  (pre-1989)  and  post  Cold  War. 

•  Large  schedule  overruns  exceeded  large  cost  overruns  as  shown  in  Figure  2. 

•  In  Figure  2,  if  the  nine  programs  whose  cost  or  schedule  overrun  was  greater  than 
100  percent  are  considered  outliers,  and  excluded,  the  average  cost  overrun  is  19 
percent  and  average  schedule  overrun  is  34  percent,  approximately  half  the  total 
averages. 

•  Figure  5  indicates  the  cost  growth  of  DoD  weapon  systems  in  their  development  phase 
compares  favorably  with  other  Governmental  and  private  industry  complex  projects 
development. 

•  The  average  duration  of  the  EMD  phase  was  shortest  during  the  Cold  War  period, 
(pre-‘89),  at  6.5  years.  The  average  duration  was  longest,  8.7  years,  during  ‘93-96.  In 
‘97  two  programs  had  an  average  duration  of  8.3  years. 

•  Excellent  cost  data  correlation  existed  between  the  total  program  costs  obtained  from 
the  “Blue  Books”  used  in  the  original  research  and  the  surrogate  total  program  costs 
obtained  from  the  SARs. 

•  The  software  complexity  of  a  program  usually  cannot  be  obtained  from  the  SAR 
reports. 

•  It  Is  too  early  to  measure  the  success  of  AR.  However  this  research  provides  an  esti¬ 
mate  of  the  DoD  acquisition  management  success  prior  to  AR  initiatives,  and  at  some 
future  date  can  be  used  to  similarly  measure  the  improvements  achieved. 

•  This  research  makes  no  attempt  to  evaluate  the  program’s  success  In  its  operational 
role. 

Operational  Test  Metrics 

•  The  number  of  test  articles  used  in  EMD  is  not  a  valid  indicator  of  the  success  of  that 
program.  The  number  of  test  hours  during  EMD  would  be  a  better  metric,  but  this  was 
not  obtainable  by  this  research. 

•  This  research  evaluation  indicates  a  noticeable  difference  between  performance 
evaluations  by  the  OTAs  and  the  independent  evaluations  of  the  DOT&E. 

•  The  OTA  operational  test  reports  are  usually  more  positive  than  the  DOT&E 
evaluations. 


TABLE  1.  Primary  Findings 
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General  Management  Metrics 


Cost  and  Schedule  Overruns.  Figure  2  is  an  updated  version  of  the  same  figure  that  appeared 
in  the  original  research  report.  It  shows  cost  and  schedule  overruns  in  the  form  of  the  ratio  of 
actual  cost  or  schedule  to  planned  cost  or  schedule.  A  program  that  completed  EMD  essentially 
on  cost  and  on  schedule  would  have  a  ratio  of  1: 1  for  both  cost  and  schedule.  Hence  the  smaller 
dotted  line  square  in  Figure  2  is  the  loci  of  all  programs  that  came  in  on  cost  and  on  schedule.  Data 
points  inside  this  square  indicate  programs  that  under  ran  cost  and  schedule.  The  larger  dotted 
line  square  is  the  loci  of  all  programs  with  a  ratio  of  2: 1 ;  that  is,  programs  that  completed  EMD 
with  exactly  a  100  percent  overrun  in  cost  and  in  schedule.  To  convert  a  ratio  reading  from  Figure 
2  into  the  applicable  overrun,  subtract  1,  and  multiply  by  100.  For  the  33  programs  contained  in 
Figure  2,  the  average  cost  overrun  was  40  percent  and  the  average  schedule  overrun  was  62 
percent  for  the  entire  period  from  1980  to  1996. 
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FIGURE  2.  Ratio  of  Actual  to  Planned  EMD  Cost  and  Schedule 

As  indicated  earlier,  some  valuable  insight  into  trends  and  the  dynamics  of  the  DoD  acquisi¬ 
tion  system  can  be  obtained  by  recording  this  data  in  three  separate-year  groups,  and  to  a  lesser 
degree,  by  individual  Service.  Figure  3  shows  the  cost  and  schedule  overrun  averages  for  the 
three-year  groups  used  in  this  report.  An  initial  observation  is  that  cost  overruns  were  the  great¬ 
est,  and  schedule  overruns  the  least  during  the  Cold  War  years  ending  in  1989.  Since  then,  eost 
overruns  have  been  less  than  schedule  overruns.  This  cost  overrun  trend  agrees  with  Rich  and 
Dews’  findings  for  programs  in  the  1960s,  although  our  schedule  overrun  figure  is  higher.  Figure 
4  shows  that  U.S.  Army  programs  had  less  cost  and  schedule  overruns  than  the  other  Serviees.  A 
superficial  possible  reason  for  this  is  the  general  perception  that  Army  equipment  is  less  complex 
than  the  other  Services. 
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FIGURE  3.  Average  Cost  and  Schedule  Overruns  by  Year  EMD  Phase  Ended 


FIGURE  4.  Average  Cost  and  Schedule  Overruns  by  Service 


DoD  average  cost  and  schedule  overruns  of  40  percent  and  62  percent  at  first  glance  may  not 
appear  to  be  good.  However,  when  you  factor  in  complexity  of  the  development,  and  schedule 
pressures  this  in  fact  may  not  be  too  bad.  In  fact,  when  compared  to  other  major  projects,  DoD 
acquisition  cost  management  is  among  the  best.  Figure  5  shows  DoD  major  weapon  systems  to  have 
a  cost  overrun  of  approximately  38  percent,  close  to  our  40  percent  figure. 

Lastly,  if  cost  and  schedule  overruns  are  looked  at  in  Total  Quality  Management  (TQM)  terms, 
and  if  100  percent  overruns  are  considered  the  statistical  process  control  acceptable  limits  of  a  DoD 
acquisition  system  under  control,  the  average  cost  overrun  is  halved  to  19  percent  and  the  average 
schedule  overrun  drops  to  34  percent.  That  is,  we  eliminate  the  nine  programs  that  had  cost  or 
schedule  overruns  of  over  100  percent.  Programs  over  the  acceptable  process  control  limits  could 
be  handled  by  some  other  extraordinary  process.  Our  research  methodology  allows  the  control 
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FIGURE  5.  Cost  Growth  In  Major  Program  Projects  (TASC)** 
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limits  to  be  set  at  any  acceptable  level  and  the  resultant  average  cost  and  schedule  overruns  can  be 
obtained. 

HMD  Duration.  The  original  research  report  indicated  that  for  24  programs  completing  EMD 
between  the  years  1980  and  1993,  the  average  duration  of  the  EMD  phase  was  7.4  years.  The  most 
recent  addition  to  this  data  shows  that  for  37  programs  (including  the  original  24),  completing  EMD 
between  1980  and  1997,  the  average  duration  of  the  EMD  phase  was  7 .8  years.  This  summary  data 
can  be  broken  down  into  year  groups  as  well  as  planned  and  actual  duration,  and  the  ratio  of  actual 
to  planned  duration.  The  latter  may  be  a  rough  estimate  of  the  realism  of  the  original  program 
planning.  Table  2  shows  this  data. 


Year  EMD  Completed 

Average  EMD  Duratic 

)n 

Planned 

Actual 

Ratio 

1980-1988  (n=12) 

4.2  yrs 

6.5  yrs 

1.6 

1989-1992  (n=11) 

4.9  yrs 

8.0  yrs 

1.7 

1993-1996  (n=121 

5.8  vrs 

8.7  vrs 

1.6 

1997  (n  =  1) 

5.0  yrs 

7.9  yrs 

1.6 

TABLE  2.  Average  EMD  Duration 

■Snr,r.p,ss  Ratings.  Now  we  turn  our  attention  to  the  inverse;  cost,  schedule  and  performance 
success  rather  than  overruns.  Table  3  contains  the  five  success  ratings  used  in  this  research.  For  each 
success  rating  level  we  indicate  the  number  of  programs  in  our  spreadsheet  that  achieved  that  suc¬ 
cess  rating  in  cost,  in  schedule,  and  in  performance.  For  the  performance  rating  we  are  using  the 
success  rating  assigned  to  that  program’s  DOT&E  BLRIP  report.  This  report  goes  directly  to  the 
Secretary  of  Defense  and  to  the  congressional  committees  concerned  with  DoD  acquisition.  In 
earlier  times  it  was  thought  of  as  the  “final  exam.”  The  average  success  ratings  shown  for  the  entire 
period  1980-1996,  indicate  performance  is  always  rated  highest  and  schedule  is  always  rated  lowest 
in  success  ratings.  It  seems  reasonable  to  expect  a  high  performance  success  rating  since,  after  all,  if 
the  system  did  not  essentially  meet  specifications  it  would  not  be  accepted.  Figure  6  (depicting  the 
average  success  ratings  by  year  the  EMD  phase  ended)  shows  again  that  during  the  Cold  War  years 
schedule  adherence  was  greater  than  more  recent  years.  And,  cost  success  has  been  greater  m  the 
post  Cold  War  period.  Figure  7  shows  success  ratings  by  Service.  Relatively  equal  cost  success 
ratings  are  shown  for  all  Services.  The  Air  Force  appears  to  have  the  least  schedule  success  and  the 
greatest  performance  success. 
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Program  Success  Ratine 


5.  Very  Successful  (V.S.) 

4.  Successful  (S.) 

3.  Fairly  Successful  (F.S.) 

2.  Marginally  Successful  (M.S.) 
1.  Not  Successful  (N.S.) 


Cost 

Success 

Rating 


Schedule 

Success 

Rating 


Performance 

Success 

Rating* 


Average  Success  Rating  S  3.3 


*Performance  Success  Rating  is  the  DOT&E  Overall  Rating 


TABLE  3.  Cost,  Schedule  and  Performance  Success  Ratings 
(Total  Number  of  Programs  Receiving  Each  Rating) 


Schedule 


Performance* 


Him 


IMMillMIli 


1980-1988  1989-1992  1993-1996  1980-1988  1989-1992  1993-1996  1980-1988  1989-1992  1993-1996 

(n=12)  (n*11)  (n=10)  (n=12)  (n=11)  (n=12)  (n=10)  (n=11)  (n=17) 

*Performance  Success  Rating  is  the  DOT&E  Overall  Rating 


FIGURE  6.  Average  Cost,  Schedule  and  Performance* 
Success  Rating  by  Year  EMD  Phase  Ended 
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FIGURE  7.  Average  Cost,  Schedule  and  Performance*  Success  Rating  by  Service 

In  our  methodology  we  have  assigned  5  points  maximum  for  a  Very  Successful  rating  in  cost  m 
schedule,  and  in  performance.  For  the  performance  rating  we  are  using  the  success  rating  ass  g 
to  that  program’s  DOT&E  BLRIP  report.  Thus,  a  “perfectly”  successful  program  would  achieve 
total  rating  score  of  15.  A  not  successful  program  would  achieve  the  minimum  score  possible  of  3. 
H“hows  .he  number  of  programs  on  .he  Y  axis  to  have  achieved  fte  var.ous  .0  al  SUC..SS 
ratings:  3  through  15  on  the  X  axis.  On  first  glance  the  majority  of  the  programs  have  achieved 
success  ratings  of  9  or  more. 


FIGURE  8.  Overall  EMD  Success  Rating 
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In  our  methodology  we  have  assigned  5  points  maximum  for  a  Very  Successful  rating  in 
cost,  in  schedule,  and  in  performance.  For  the  performance  rating  we  are  using  the  success  rating 
assigned  to  that  program’s  DOT&E  BLRIP  report.  Thus,  a  “perfectly”  successful  program  would 
achieve  a  total  rating  score  of  15.  A  not  successful  program  would  achieve  the  minimum  score 
possible  of  3.  Figure  8  shows  the  number  of  programs  on  the  Y  axis  that  have  achieved  the  various 
total  success  ratings:  3  through  15  on  the  X  axis.  On  first  glance  the  majority  of  the  programs  have 
achieved  success  ratings  of  9  or  more. 

Baselining  Acquisition  Reform.  The  methodology  used  in  this  research  can  be  one  method 
used  to  evaluate  the  effectiveness  of  the  AR  initiative.  The  process  would  consist  of  three  steps. 
First,  select  the  date  that  AR  first  was  effective  in  the  field,  and  measure  the  cost,  schedule  and 
performance  success  levels  of  DoD  acquisition  at  that  time.  Second,  over  a  period  of  time  intro¬ 
duce  the  AR  improvements/changes  into  the  DoD  acquisition  system.  And  finally,  at  some  appro¬ 
priate  later  time  remeasure  the  cost,  schedule  and  performance  success  of  the  then  DoD  acquisi¬ 
tion  system. 

The  first  step  has  already  been  accomplished  and  is  completely  detailed  in  an  Acquisition 
Review  Quarterly  artiele  approved  for  publication  around  January  2000.  That  article  suggests  that 
the  date  AR  was  first  effective  “in  the  workplace”  was  July,  1996,  and  at  that  time  the  DoD  acquisi¬ 
tion  system,  over  the  prior  four  years,  had  an  average  cost  overrun  of  46  percent  and  an  average 
schedule  overrun  of  81  percent.  Using  this  research  success  ratings  of  5  down  to  1,  and  showing 
percent  equivalents  we  have  the  pre-AR  baseline: 

Parameter  Success  Rating  and  Percent  Equivalent 


Cost 

3.5 

(70  percent) 

Schedule 

2.1 

(41  percent) 

Performance 

3.8 

(76  percent) 

In  creating  the  database,  it  became  clear  that  its  usefulness  in  analyzing  programs  required 
that  other  parameters  beyond  cost,  schedule  and  performance  would  be  desirable.  To  identify  what 
characteristics  correlated  with  success  in  EMD,  or  with  failure,  possible  causes  had  to  be  included  in 
the  collected  data.  In  particular,  six  programmatics  were  included. 

Average  Tenure  of  the  Program  Manager.  It  has  been  held  for  some  time  that  programs 
would  be  more  in  control  if  program  managers  were  stabilized  in  position.  To  be  able  to  examine 
this,  the  number  of  program  managers  during  EMD  was  recorded.  Average  tenure  was  calculated  by 
dividing  the  actual  duration  of  EMD  by  the  number  of  program  managers  that  program  had  during 
EMD.  This  method  of  analysis  results  in  an  average  PM  tenure  that  is  lower  than  the  actual  PM 
tenure  over  the  entire  life  of  the  program.  Our  average  PM  tenure  for  the  34  programs  was  31 
months.  Ron  Fox  in  his  book'®  states  “...PMs  averaged  approximately  27  months  on  a  given  pro¬ 
gram  as  either  the  program  manager  or  deputy  program  manager.”  Also  General  Lawrence  Skantze, 
Commander  of  the  Air  Force  Systems  Conunand,  is  quoted  as  saying  Air  Force  PMs  average  29 
months  in  the  job.  Our  research  indicates  the  average  PM  tenure  now  is  over  31  months,  but  we 
could  find  no  correlation  between  PM  tenure  and  program  EMD  success. 
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Type  of  Contract.  There  are  many  opinions  about  what  type  of  EMD  contract  is  most  likely 
to  promise  successful  completion.  To  include  this  in  any  future  analysis,  the  type  of  contract  awarded 
for  EMD  was  recorded.  Firm  Fixed  Price  (FFP),  Fixed  Price  Incentive  Fee  (FPIF)  and  Cost  Plus 
Incentive  Fee  (CPIF)  had  overall  average  EMD  program  success  ratings  of  9.2,  9.4,  and  11.5  re¬ 
spectively  out  of  a  maximum  of  15  points.  Cost  Plus  Award  Fee  (CPAF)  programs  were  conspicu¬ 
ously  less  successful  at  6.6  points. 

Competition.  The  pros  and  cons  of  having  competition  during  EMD  have  been  argued  for 
years.  To  assist  in  this  discussion,  the  presence  or  absence  of  competition  in  EMD  was  recorded. 
Programs  that  had  competition  in  EMD  achieved  an  average  program  success  rating  of  8.6  while 
programs  that  had  no  competition  in  EMD  had  a  success  rating  of  9.9.  However,  the  spread  of  high 
and  low  ratings  in  both  cases,  as  well  as  other  confounding  concerns  requires  further  analysis. 

Number  of  Associate  Contractors.  It  is  commonly  held  that  the  complexity  of  a  program, 
and  therefor  the  likelihood  of  encountering  difficulties,  is  related  to  the  number  of  associated  con¬ 
tractors.  To  be  able  to  include  this  parameter  in  analysis,  this  number  for  EMD  was  collected.  Our 
research  looked  at  27  programs  that  utilized  from  zero  to  eleven  associate  contractors.  With  one 
exception,  all  programs  achieved  relatively  high  program  success  ratings  regardless  of  the  number  of 
associate  contractors  used. 

Number  of  Production  Quantity  Changes.  It  is  generally  agreed  that  the  more  programmatic 
changes  there  are  in  a  program,  the  more  likely  it  is  that  the  program  will  experience  difficulties.  One 
quantity  which  often  reflects  program  success  or  failure  is  the  number  of  production  items.  Changes 
in  this  quantity  can  reflect  budget  decisions,  performance  in  testing,  or  need  for  the  item  in  the  field. 
In  order  to  include  this  as  a  reflection  of  program  difficulty,  the  number  of  changes  in  production 
quantity  was  recorded.  No  meaningful  conclusion  could  be  drawn  from  analysis  of  this  parameter. 

.SAR  percent  of  Exception/Annual.  Congress  has  established  the  SAR  reporting  require¬ 
ments  to  include  both  routine  (annual)  and  exception  (breaches  of  cost  or  schedule  baselines).  The 
intent  of  exception  reporting  is  to  identify  programs  requiring  special  oversight.  In  order  to  analyze 
the  success  of  this  reporting  requirement,  it  would  be  interesting  to  find  if  there  is  a  correlation 
between  the  success  of  the  program  and  the  frequency  of  exception  SAR  reporting.  Therefore, 
collected  data  included  the  number  of  annual  S  ARs  during  EMD  and  the  number  of  exception  S  ARs 
during  EMD.  Since  the  analysis  of  this  parameter  should  eliminate  the  variable  of  duration  of  EMD, 
the  percent  of  exception  SARs  to  annual  SARs  was  calculated  and  included  in  the  database.  We 
discovered  in  the  ‘80s  several  programs  submitted  quarterly  SARs  on  a  routine  basis  and  our  re¬ 
search  to  date  did  not  distinguish  between  required  exceptional  SARs  and  routine  quarterly  SARs. 
For  this  reason,  and  others,  no  comment  is  made  at  this  time. 

Operational  Test  Metrics 

Test  Articles  and  Program  Success.  The  original  objective  of  this  research  was  to  determine  the 
relationship,  if  any,  between  the  number  of  test  articles  used  in  EMD  and  the  success  of  that  program 
in  EMD.  The  original  research  report  stated  there  was  a  tenuous  positive  correlation  and  went  on  to 
explain  the  many  nuances  and  uncertainties  associated  with  the  relevant  raw  data.  The  current  research 


attempted  to  continue  obtaining  data  in  this  area  but  uncertainties  continued.  For  example,  more  recent 
SAR’s  contain  the  data  “NFC,”  not  fully  configured.  The  presumption  is  they  are  units  procured  with 
engineering  funds  and  used  for  testing,  but  this  cannot  be  proven.  Lastly,  it  was  realized  that  the 
number  of  test  articles  was  not  as  good  a  metric  as  the  number  of  test  hours.  As  a  result,  the  conclusion 
is  that  the  original  and  current  research  cannot  provide  an  answer  to  the  question. 

Comparative  Operational  Test  Data.  The  following  sections  discuss  the  comparison  between  the 
performance  success  ratings  assigned  to  test  reports  issued  by  the  Service  OTAs  and  the  performance 
success  rating  assigned  to  the  independent  BLIP  evaluation  report  of  the  DOT&E.  Both  OTA  test 
reports  and  DOT&E  BLRIP  evaluations  discuss  and  evaluate  Operational  Effectiveness  and  Opera¬ 
tional  Suitability.  Both  generally  have  an  overview/conclusions  section.  Therefore  in  our  research  we 
have  assigned  a  success  rating  (5  down  to  1),  for  operational  effectiveness,  operational  suitability,  and 
a  subjective  third  rating  for  Overall  Effectiveness.  This  latter  rating  attempts  to  evaluate  and  record  the 
executive  comments  usually  found  in  operational  test  reports  or  evaluations.  Thus  the  maximum  suc¬ 
cess  rating  for  an  OTA  report  or  a  DOT&E  evaluation  is  15;  a  score  of  3  is  minimum.  This  comparison 
between  OTA  and  DOT&E  is  considered  to  be  of  interest  to  the  management  of  both  organizations. 
Also  it  provides  data  to  resolve  the  perception  a  few  years  ago  that  “DOT&E  routinely  rubber  stamps 
the  results  of  lOT&Es.” 

Figure  9  shows  the  number  of  times/programs  that  OTA  ratings  agreed  or  disagreed  with  DOT&E 
evaluations.  For  example,  out  of  34  programs,  there  were  5  times/programs  where  the  OTA  perfor¬ 
mance  success  rating  was  exactly  equal  to  the  DOT&E  evaluation.  The  predominant  rating,  in  10 
times/programs  was  a  difference  of  2  success  rating  points  between  OTA  and  DOT&E.  With  equal 
ratings  in  only  5  of  31  instances,  and  with  a  surprising  difference  of  7  rating  points  (out  of  15)  in  three 
cases,  it  seems  difficult  to  say  “DOT&E  rubber  stamps  OTA  reports.”  Figure  10  looks  at  the  same 
difference  in  ratings  by  year  the  EMD  phase  ended.  Figure  1 1  looks  at  the  data  by  individual  Service. 
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FIGURE  10.  Analysis  of  the  Average  Difference  Between  OTA/DOT&E 
Performance  Success  Ratings— By  Year  EMD  Phase  Ended 


FIGURE  11.  Analysis  of  the  Average  Difference  Between  OTA/DOT&E 
Performance  Success  Ratings— By  Service 


To  date  we  have  discussed  relative  differences  between  the  success  ratings  the  research  team 
has  assigned  to  lOT&E  reports  and  BLRIP  evaluations.  What  is  not  visible  is  which  organization 
does  the  research  team  feels  rates  higher  or  lower?  Figures  1 2- 1 8  are  designed  to  show  this  interest¬ 
ing  data.”  For  the  entire  period,  1980-1996,  Figure  12  shows  the  relative  success  ratings  assigned 
by  the  research  team  to  OTA  reports  and  DOT&E  evaluations.  The  diagonal  line  indicates  equal 
success  ratings  assigned  to  a  program’s  OTA  and  DOT&E  evaluations.  A  data  point  above  the  line 
indicates  the  research  team  evaluated  the  OTA  report  more  highly  (positive)  than  the  DOT&E  evalu¬ 
ation.  Below  the  line  the  reverse  is  true.  An  impression  of  Figure  12  is  that  the  OTA  reports  were 
generally  rated  more  positive.  Also,  where  the  DOT&E  reports  were  rated  more  positive,  the  ratings 
were  all  at  the  upper  range  of  the  scale;  11  or  more.  Figures  13-15  show  the  same  data  by  year  the 
EMD  phase  ended.  In  Figure  14  for  the  time  period  1989-1992,  all  data  indicates  the  research  team 
evaluated  the  OTA  reports  to  be  more  positive  than  the  DOT&E  evaluations.  Figures  16-18  indicate 
the  same  data  by  Service  with  no  significant  difference  discemable. 


FIGURE  12.  Analysis  of  OTA/DOT&E  Performance  Success  Ratings 
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FIGURE  13.  Analysis  of  OTA/DOT&E  Performance  Success  Ratings 
for  Programs  whose  EMD  Phase  Ended  between  1980-88 


FIGURE  14.  Analysis  of  OTA/DOT&E  Performance  Success  Ratings 
for  Programs  whose  EMD  Phase  Ended  between  1989-92 
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FIGURE  15.  Analysis  of  OTA/DOT&E  Performance  Success  Ratings 
for  Programs  whose  EMD  Phase  Ended  between  1993-96 


FIGURE  16.  Analysis  of  OTA/DOT&E  Performance 
Success  Ratings  for  Programs  for  Service— Army 
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FIGURE  17.  Analysis  of  OTA/DOT&E  Performance 
Success  Ratings  for  Programs  for  Service-Navy 
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FIGURE  18.  Analysis  of  OTA/DOT&E  Performance 
Success  Ratings  for  Programs  for  Service — Air  Force 
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CHAPTER  6 

CONCLUSIONS  AND  RECOMMENDATIONS 


Conclusions 

•  This  trend  data  is  informative  and  can  be  used  to  focus  managerial  change  efforts.  The  end  of 
the  Cold  War  (1989)  appears  to  have  had  a  significant  impact  on  cost  and  schedule  data. 

•  The  DoD  1980-1996  average  EMD  cost  and  schedule  overruns  were  40  percent  and  62  per¬ 
cent  respectively.  This  compares  favorably  with  other  government  and  private  complex  devel¬ 
opment  projects. 

•  The  duration  of  the  EMD  phase  has  increased  since  1980-88,  but  in  1997  the  only  currently 
entered  data  shows  a  reversal  trend. 

Recommendations 

•  Review  the  SAR  format  and  required  contents.  The  preparation  instructions  for  SARs  state 
“The  results  of  operational  testing  will  take  highest  precedence  in  determination  of  demon¬ 
strated  performance.”  However,  there  is  little  or  no  requirement  to  include  OT  periods  in  the 
SAR  schedule  section.  The  sample  SAR  included  in  the  DoD  instructions  require  reporting 
only  “DT/OT II  Complete.”  Also,  the  degree  of  software  required  within  the  developing  sys¬ 
tem  is  an  important  consideration  affecting  program  success.  The  SAR  requires  no  specific 
comments  about  software. 

•  SARs  are  required  for  ACAT I  programs  only.  A  report  similar  to  the  SAR  should  be  required 
for  less  than  ACAT  I  programs  listed  on  the  DOT&E  Oversight  List. 

•  This  research  data  is  contained  on  an  Excel  spreadsheet  which  limits  analysis.  A  prototype 
advanced  database  has  been  developed  to  allow  for  multi- variate  regression  analyses,  but  is  not 
funded  at  the  present  time. 

•  This  factual  data  should  be  combined  with  other  databases,  similar  research  efforts,  and  exist¬ 
ing  analyses  to  develop  “leading  indicators”  that  might  be  used  to  predict  program  develop¬ 
mental  success.  This  would  include  a  more  sophisticated  database  for  this  research  data. 

•  Research  should  be  initiated  to  compare  and  contrast  the  results  of  the  ACTD  initiative  with 
the  results  of  other  systems  being  developed  under  the  5000  series  regulations. 

At  some  future  date,  the  success  of  the  current  AR  initiative  should  be  measured  using  the 

methodology  and  baseline  developed  by  this  research  effort. 
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END  NOTES 


'The  U.S.  Navy  and  U.S.  Marine  Corps  call  this  specific  phase  of  Operational  Testing  (OT),  Opera¬ 
tional  Evaluation  (OPEVAL).  In  prior  years  the  Office  of  the  Secretary  of  Defense  (OSD)  called  this 
particular  test  the  “final  dedicated  phase  of  operational  testing  prior  to  Milestone  (MS)  III.” 

^United  States  Code,  Title  10,  Section  2399. 

^Conversations  between  Dr.  E.  Seglie,  Science  Advisor  to  the  Director  of  Operational  Test  and 
Evaluation  (DOT&E)  and  Professor  R.  Reig,  Defense  Systems  Management  College  (DSMC). 

‘‘Gailey,  C.K.,  Reig,  R.W.,  Weber,  W.,  and  George,  L.R.  (1995).  A  Study  of  the  Relationship  Be¬ 
tween  Initial  Production  Test  Articles  used  in  a  System  Development  Program  and  the  Success  of 
that  Program  (The  DSMC  Press,  TR2-95)  Fort  Belvoir,  VA. 

^Each  Service  has  an  Operational  Test  Activity  whose  Commander  reports  directly  to  the  military 
chief  of  that  Service.  The  OTA  is  independent  of  the  Program  Manager  and  the  Developing  Agency. 

^Department  of  Defense  (1991).  Defense  Acquisition  Management  Documentation  and  Reports, 
Part  17,  Selected  Acquisition  Report  (DoD  5000.2-M).  Washington,  DC. 

’Rich,  M.  &  Dews,  E.  (1986).  Improving  the  Military  Acquisition  Process  (The  RAND  Corpora¬ 
tion,  R-3373-AF/RC,  p.  6).  Santa  Monica,  CA. 

*Drezner,  J.  A.,  et  al.  (1993).  An  Analysis  of  Weapon  System  Cost  Growth  (The  RAND  Corporation, 
MR-29 1-AF,  p.  xiii).  Santa  Monica,  CA. 

^Gailey,  C.K.,  Reig,  R.W.,  Weber,  W.,  and  George,  L.R.  (1995).  A  Study  of  the  Relationship  Be¬ 
tween  Initial  Production  Test  Articles  used  in  a  System  Development  Program  and  the  Success  of 
that  Program  (The  DSMC  Press,  TR2-95)  Fort  Belvoir,  VA. 

'"Wiles,  J.A.  (25  February  1994).  Study  Group  Report  on  Evaluation  of  Electronic  System  Acquisi¬ 
tion,  Report  to  the  Under  Secretary  of  Defense  (Acquisition  and  Technology).  Washington,  DC. 

"Packard,  D.  (30  June  1986).  A  Quest  for  Excellence.  Final  report  to  the  President  by  the  President’s 
Blue  Ribbon  Commission  on  Defense  Management,  p.  xxiii.  Washington,  DC. 

'’Mayer,  K.  R.  (1993).  The  Development  of  the  Advanced  Medium-Range  Air-to-Air  Missile:  A 
case  study  of  Risk  and  Reward  in  Weapon  System  Acquisition  (The  RAND  Corporation,  N-3620- 
AF,  p.  vi). 

'^Fox,  J.  Ronald.  The  Defense  Management  Challenge:  Weapons  Acquisition.  Boston:  Harvard 
Business  School  Press,  1988,  p.  64. 

^^Ibid,  p.  72. 
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^Hbid,  p.  142. 

*®Fox,  p.  64. 

'^We  are  indebted  to  Dr.  E.  Seglie,  Science  Advisor  to  the  DOT&E  for  this  display  concept. 
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APPENDIX  A 
DEFINITIONS 


Acquisition  Category  (ACAT) 

ACAT I  -  programs  are  Major  Defense  Acquisition  Programs  (MDAPs).  An  MDAP  is  de¬ 
fined  as  a  program  estimated  by  the  Under  Secretary  of  Defense  (Acquisition  and  Technol¬ 
ogy)  (USD(A&T))  to  require  eventual  expenditure  for  research,  development,  test,  and  evalu¬ 
ation  of  more  than  $355  million  (fiscal  year  (FY)  96  constant  dollars)  or  procurement  of 
more  than  $2,135  billion  (FY96  constant  dollars),  or  those  designated  by  the  USD(A&T)  to 
be  ACAT  I. 

ACAT  II  -  programs  are  defined  as  those  acquisition  programs  that  do  not  meet  the  criteria 
for  an  ACAT  I  program,  but  do  meet  the  criteria  for  a  major  system.  A  major  system  is 
defined  as  a  program  estimated  by  the  DoD  Component  Head  to  require  eventual  expendi¬ 
ture  for  research,  development,  test,  and  evaluation  of  more  than  $135M  in  FY96  constant 
dollars,  or  for  procurement  of  more  than  $640M  in  FY96  constant  dollars,  or  those  desig¬ 
nated  by  the  DoD  Component  Head  to  be  ACAT  II. 

ACAT  III  -  programs  are  defined  as  those  acquisition  programs  that  do  not  meet  the  criteria 
for  an  ACAT  I,  or  an  ACAT  II.  The  MDA  is  designated  by  the  CAE  and  shall  be  at  the  lowest 
appropriate  level.  This  category  includes  less-than-major  AISs. 

ACAT  IV  (Army  only)  -  ACAT  programs  in  the  Army  not  otherwise  designated  as  ACAT 
I,  II  or  III  are  designated  ACAT  IV.  ACAT  IV  programs  are  managed  by  a  systems  man¬ 
ager  within  a  materiel  command  as  opposed  to  ACAT  I-III  programs  which  are  managed 
by  a  PM. 

ACAT  IV  (Navy  and  Marine  Corps  only)  -  ACAT  programs  in  the  Navy  and  Marine 
Corps  not  otherwise  designated  as  ACAT  I,  II  or  III  are  designated  ACAT  IV.  There  are  two 
categories  of  ACAT  IV  programs:  IVT  and  IVM.  ACAT  IVT  programs  require  operational 
test  and  evaluation  while  ACAT  IVM  programs  do  not. 

Advanced  Concept  Technology  Demonstration  (ACTD)  -  A  means  of  demonstrating  mature 
technology  to  address  critical  military  needs.  ACTDs  themselves  are  not  acquisition  programs, 
but  are  designed  to  provide  a  residual,  usable  capability  upon  completion,  and/or  transition  into 
acquisition  programs.  Funding  is  programmed  to  support  two  years  in  the  field.  ACTDs  are  funded 
with  Advanced  Technology  Development  (ATD)  funds. 

Acquisition  Reform  -  An  ongoing  series  of  initiatives  sponsored  by  OSD  (especially  USD(A&T) 
and  DUSD(AR))  to  streamline  and  tailor  the  acquisition  process.  Initiatives  include  statutory  and 
regulatory  reform,  CAIV,  reform  of  specifications  and  standards  policy,  preference  for  commer¬ 
cial  items,  electronic  data  interchange  and  the  use  of  the  IPPD/IPT  management  philosophy  for 
systems  development  and  oversight. 
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Beyond  Low  Rate  Initial  Production  (BLRIP)  -  (See  LRIP  below).  When  a  developing  system 
seeks  a  decision  to  go  into  full  rate  production  (FRP),  the  DOT&E  must  provide  a  report  to  Congress 
and  the  Secretary  of  Defense.  This  report  is  commonly  called  the  “Beyond  LRIP”  report. 

Blue  Books  -  The  name  commonly  given  to  the  books  prepared  for  the  members  of  the  Defense 
Acquisition  Board  (DAB)  prior  to  Milestone  Decision  Meetings.  They  contained  the  major  data 
such  as  planned  cost  and  schedule  for  the  system  under  review.  These  books  were  discontinued  m 

1993. 

Cost  As  an  Independent  Variable  (CAIV)  -  Methodologies  used  to  acquire  and  operate  afford¬ 
able  DoD  systems  by  setting  aggressive,  achievable  life  cycle  cost  objectives,  and  managing  achieve¬ 
ment  of  these  objectives  by  trading  off  performance  and  schedule,  as  necessary.  Cost  objectives 
balance  mission  needs  with  projected  out-year  resources,  taking  into  account  anticipated  process 
improvements  in  both  DoD  and  industry.  CAIV  has  brought  attention  to  the  government’s  respon¬ 
sibilities  for  setting/adjusting  life-cycle  cost  objectives  and  for  evaluating  requirements  in  terms  of 
overall  cost  consequences. 

Director,  Operational  Test  and  Evaluation  (DOT&E)  -  The  senior  individual  within  the  Office  of 
the  Secretary  of  Defense  responsible  for  operational  testing  policy  and  oversight.  This  position  is 
mandated  by  law,  10USC2366. 

Initial  Operational  Test  and  Evaluation  (lOT&E)  -  Operational  test  and  evaluation  conducted 
on  production,  or  production  representative  articles,  to  determine  whether  systems  are  operation¬ 
ally  effective  and  suitable,  and  which  supports  the  decision  to  proceed  beyond  low  rate  initial  pro¬ 
duction  (LRIP).  The  Navy  calls  this  test  OPEVAL,  Operational  Evaluation. 

Integrated  Product  and  Process  Development  (IPPD)  -  A  management  technique  that  simulta¬ 
neously  integrates  all  essential  acquisition  activities  through  the  use  of  multidisciplinary  teams  to 
optimize  the  design,  manufacturing,  and  supportability  processes.  IPPD  facilitates  meeting  cost  and 
performance  objectives  from  product  concept  through  production,  including  field  support.  One  of 
the  key  IPPD  tenets  is  multidisciplinary  teamwork  through  Integrated  Product  Teams  (IPTs). 

Integrated  Product  Team  (IPT)  -  Team  composed  of  representatives  from  appropriate  functional 
disciplines  working  together  to  build  successful  programs,  identify  and  resolve  issues,  and  make 
sound  and  timely  recommendations  to  facilitate  decision  making.  There  are  three  types  of  I^s. 
overarching  IPTs  (OIPTs)  focus  on  strategic  guidance,  program  assessment,  and  issue  resolution; 
working  level  IPTs  (WIPTs)  identify  and  resolve  program  issues,  determine  program  status,  and 
seek  opportunities  for  acquisition  reform;  and  program  level  IPTs  focus  on  program  execution  and 
may  include  representatives  from  both  government  and  after  contract  award  industry. 

Low-Rate  Initial  Production  (LRIP)  -  The  minimum  number  of  systems  (other  than  ships  and 
satellites)  to  provide  production  representative  articles  for  operational  test  and  evaluation  (OT&E), 
to  establish  an  initial  production  base,  and  to  permit  an  orderly  increase  in  the  production  rate 
sufficient  to  lead  to  full-rate  production  upon  successful  completion  of  operational  testing.  For 
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major  defense  acquisition  programs  (MDAPs),  LRIP  quantities  in  excess  of  10  percent  of  the 
acquisition  objective  must  be  reported  in  the  selected  acquisition  report  (SAR).  For  ships  and  satel¬ 
lites  LRIP  is  the  minimum  quantity  and  rate  that  preserves  the  mobilization  base. 

Milestone  Decision  Authority  (MDA)  -  The  individual  designated  in  accordance  with  criteria 
established  by  the  USD(A&T),  or  by  the  Assistant  Secretary  of  Defense  (Command,  Control,  Com¬ 
munications,  and  Intelligence)  (ASD  (C^I))  for  automated  information  system  (AIS)  acquisition 
programs,  to  approve  entry  of  an  acquisition  program  into  the  next  phase  (DoD  5000.2-R). 

Operational  Test  Activities  (OTAs)  -  Each  Service  organizationally  has  an  independent  activity 
charged  with  the  conduct  and  evaluation  of  operational  testing  within  that  Service.  The  Conunander 
of  the  OTA  is  usually  a  two-star  General  or  Flag  officer. 

Selected  Acquisition  Reports  (SAR)  -  Standard,  comprehensive,  summary  status  reports  on  ma¬ 
jor  defense  acquisition  programs  (MDAPs)  (Acquisition  Category  (ACAT)  I)  required  for  periodic 
submission  to  the  Congress.  They  include  key  cost,  schedule,  and  technical  information. 

DoD  Directive  (DoDD)  5000.1  “Defense  Acquisition”  -  The  principal  DoD  directive  on  acquisi¬ 
tion,  it  states  policies  and  principles  for  all  DoD  acquisition  programs  and  identifies  the  Department’s 
key  acquisition  officials  and  forums. 

DoD  Regulation  5000.2-R  “Mandatory  Procedures  for  Major  Defense  Acquisition  Programs 
(MDAPs)  and  Major  Automated  Information  System  (MAIS)  Acquisition  Programs”  -  Sets 
forth  mandatory  procedures  for  MDAPs  and  MAISs  and,  specifically  where  stated,  for  other  than 
MDAPs  or  MAISs.  Authorizes  milestone  decision  authorities  (MDAs)  to  tailor  procedures  as  they 
see  fit,  consistent  with  statutory  requirements. 
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APPENDIX  B 

CORRELATION  OF  COST  DATA  SOURCES 


Cost  Success  as  measured  under  the  previous  method, 
compared  to  Cost  Success  as  measured  under  the  present  method. 


Previous  method  (as  used  in  the  1995  Technical  Report)  collected  all  the  costs  identifiable  to 
EMD. 

Present  method  collects  only  RDT&E  Appropriation  costs  identifiable  to  EMD. 

Cost  Success  assigns  a  rating  (5  to  1)  based  on  how  close  actual  costs  were  to  plan. 

Rationale  for  change:  using  all  EMD  costs  puts  too  much  weight  on  Procurement  Appropria¬ 
tion  variance,  since  Procurement  costs  generally  are  much  higher  than  RDT&E  costs.  Program  cost 
success  thus  becomes  LRIP  success. 

Program  #  Previous  Rating  Present  Rating 


8 

5 

5 

9 

5 

5 

11 

5 

5 

12 

4 

4 

13 

5 

4 

14 

5 

4 

15 

5 

4 

16 

4 

4 

17 

2 

2 

19 

3 

4 

20 

5 

5 

21 

1 

1 

22* 

5 

3 

23 

1 

1 

Programs  with  the  same  rating: 

9 

64% 

Programs  with  the  present  rating  1  higher: 

1 

7% 

Programs  with  the  present  rating  1  lower: 

3 

21% 

*Programs  whose  ratings  differ  by  2: 

1 

7% 

(this  is  B-IB,  and  the  difference  may  be  due  to  program  redefinition) 


Programs  whose  ratings  differ  by  0  or  1 :  13  93% 
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Spreadsheet  Column  Descriptions 
Notes 

Spreadsheet 
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SPREADSHEET  COLUMN  DESCRIPTIONS 


As  of  1  July  1999 
=  Computer  Generated  Item 


Primary  Spreadsheet 

rnliimn  No.  Description 

1  Program  Name 

2  Program  Number  (Chronologically  by  MS  HI  date) 

3  System  Type 

4  The  actual  date  of  the  MS  in  DAB  or  Full  Rate  Production  (FRP)  decision 

5  Notes  (see  Table  of  Notes) 

6  The  DoD  Component  or  Lead  Service 

7  The  Program  EMD  budget  success  rating  assigned  (see  the  study  criteria) 

8  The  actual  percentage  the  program  overran  the  planned  EMD  budget 

((Col  2 1  -  Col  20  divided  by  Col  20)  X  100)  * 

9  The  Program  EMD  schedule  success  rating  assigned  (see  the  study  criteria) 

10  The  actual  percentage  the  program  overran  the  planned  EMD  schedule 

((Col  17  -  Col  16  divided  by  Col  16)  X  100)  * 

1 1  The  Program  EMD  performance  success  rating  assigned  by  the  T&E  Dept. 

Subject-Matter  Expert  (SME)  Panel  after  analyzing  the  Service  Operational 
Test  Activity  (OTA)  lOT&E  or  OPEVAL  or  TER  report  following  study 
criteria.  Success  ratings  for  effectiveness,  suitability,  and  overall  success 
are  assigned. 

12  The  Program  EMD  performance  success  rating  assigned  by  the  T«&E  Dept.  SME 

Panel  after  analyzing  the  DOT&E  BLRIP  evaluation  report.  The  same  three 
success  rating  categories  used  for  OTA  reports  are  used  here. 
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Expanded  Spreadsheet 

Column  No.  Description 

13  The  actual  date  of  the  MS  n  DAB  meeting 

14  The  planned  date  (at  MS  II)  for  the  program  MS  III  DAB 

15  The  actual  date  of  the  MS  El  DAB  (or  equivalent  for  the  Full  Rate  Production 

(FRP)  decision) 

16  The  planned  duration  of  HMD  in  years  (Col  14  -  Col  13)  * 

17  The  actual  duration  of  EMD  in  years  (Col  15  -  Col  13)  * 

1 8  The  ratio  of  actual  duration  of  EMD  to  the  planned  duration 

(Col  17  divided  by  Col  16)  * 

19  The  actual  months  the  program  overran  the  planned  EMD  schedule 

(Col  17 -Col  16)* 

20  The  planned  RDT&E  cost  of  EMD  as  estimated  at  MS  II  (TY$) 

21  The  actual  RDT&E  cost  of  EMD  as  reported  at  MS  III  (TY$) 

22  The  ratio  of  the  actual  RDT&E  cost  of  EMD  to  the  planned  cost 

(Col  21  divided  by  Col  20)* 

23  The  planned  EMD  Procurement  Cost  as  reported  at  MS  n 

24  The  actual  EMD  Procurement  Cost  as  reported  at  MS  III 

25  The  total  program  Procurement  Cost  as  reported  at  MS  III 

26  Percentage  of  EMD  Procurement  Costs  to  total  Procurement  Costs 

((Col  24  divided  by  Col  25)  X  100)  * 

27  The  total  actual  program  costs  as  reported  at  MS  III  (Col  21  +  Col  25  )  * 

28  Percentage  of  EMD  costs  to  total  program  costs 

((Col  21  +  Col  24  divided  by  Col  25)  X  100  )  * 

29  The  number  of  Program  Managers  assigned  in  EMD 

30  The  average  tenure  (years)  of  PMs  assigned  during  EMD  (Col  1 7  divided  by  Col  29) 

3 1  The  number  of  annual  Selected  Acquisition  Reports  (S  ARs)  issued  covering  EMD 

32  The  number  of  Exception  S  ARs  during  EMD 

33  The  percentage  of  Exception  S  ARs  to  Annual  S  ARs 

((Col  32  divided  by  Col  31)  X  100)  * 

34  The  number  of  LRIP  systems  purchased  in  EMD  with  RDT&E  funds  and 

presumably  used  for  testing 

35  The  number  of  LRIP  systems  purchased  in  EMD  with  Procurement  funds, 

and  presumably  used  for  other  than  test  purposes 

36  The  total  LRIP  systems  purchased  in  EMD  (Col  34  +  Col  35)  * 

37  Total  planned  quantity  at  MS  II 

38  Total  actual  quantity  at  MS  IE 

39  Percent  change  in  quantity  from  MS  E  to  MS  IE 

((Col  37  -  Col  38  divided  by  Col  37)  X  100)  * 

40  Number  of  changes  to  planned  procurement  quantity  during  EMD 
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41  The  percentage  of  LRIP  RDT&E  funded  systems  to  the  total  actual  procurement 

at  MS  m  ((Col  34  divided  by  Col  38)  X  100)  * 

42  The  percentage  of  LRIP  Procurement  funded  systems  to  the  total  actual 

procurement  at  MS  HI  ((Col  35  divided  by  Col  38)  X  100 )  * 

43  The  percentage  of  total  LRIP  systems  purchased  in  EMD  to  total  actual 

procurement  at  MS  HI  ((Col  36  divided  by  Col  38)  X  100)  * 

44  The  percentage  of  RDT&E  funded  systems  to  total  EMD  systems 

((Col  34  divided  by  Col  36)  X  100 )  * 

45  The  system  is  being  developed  as  a  Multi-Service  (Joint)  program?  Yes/No 

46  Advanced  Concept  Technology  Demonstration  (ACTD)  or  a  special 

exception  program?  Yes/No 

47  The  duration  of  ACTD  in  years 

48  Yes  indicates  the  DOT&E  issued  a  “Beyond  LRIP”  report 

49  Indicates  whether  the  program  is  a  major  modification  (Yes)  or  a  new 

development  (No)  in  EMD 

50  Indicates  whether  the  SARs  or  OT  reports  record  the  number  of  Software 

Lines  of  Code  associated  with  the  program 

5 1  The  Program  Manager’s  estimate  of  (technical)  risk  as  stated  at  MS  n 

52  Indicates  whether  the  program  used  prime  contractor  competition  in  the 

Program  Definition/Risk  Reduction  (PDRR)  phase 

53  Indicates  whether  the  program  used  prime  contractor  competition  in  the 

EMD  phase  of  the  program 

54  Indicates  the  type  of  contract  used  in  EMD.  (FFP=  Firm  Fixed  Price, 

FPI  =  Fixed  Price  Incentive,  CPIF  =  Cost  Plus  Incentive  Fee, 

CPAF  =  Cost  Plus  Award  Fee) 

55  Number  of  associate  contractors  used  at  the  prime  system  level,  WBS  Level  II  or  HI 

56  Was  Early  Operational  Assessments  (EOA)  or  Operational  Assessments  (OA) 

used?  Y/N 

57  Was  combined  DT/OT  used,  and  mentioned  in  the  SARs  or  OT  reports 

58  Number  of  Critical  Operational  Issues,  Effectiveness  (COI(E))  shown  in  OT  report 

59  Number  of  Critical  Operational  Issues,  Suitability  (COI(S))  shown  in  the  OT  report 

60  The  planned  start  date  of  lOT&E/OPEVAL 

61  The  actual  start  date  of  lOT&E/OPEVAL 

62  The  planned  end  date  of  lOT&E/OPEVAL 

63  The  actual  end  date  of  lOT&E/OPEVAL 

64  The  ratio  of  the  actual  duration  of  the  lOT&E/OPEVAL  test  to  the  planned 

duration  (Col  61  to  Col  63  time  interval  divided  by  Col  60  to  Col  62  time 
interval)  * 
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NOTES 


7/1/99 


X  =  Data  not  available;  usually  program  had  no  SAR. 

N/A  =  Data  not  applicable. 

1 .  Service  lOT&E  or  OPEVAL  Report  was  not  available. 

2.  Evaluation  rating  based  on  DOT&E  Annual  Report. 

3.  Anti-SATellite  (ASAT);  MS  III  was  scheduled  for  Mar  88,  but  in  Dec  87  the  program  was 
terminated  due  to  Congressional  moratorium  on  space  testing. 

4.  TRI-service  TACtical  (TRITAC)  switch;  insufficient  data  in  the  FY86  DOT&E  Annual  Report 
to  evaluate  this  program. 

5.  This  is  the  first  MAISARC  program  subject  to  evaluation  by  DOT&E. 

6.  SAR  Production  Baseline  Estimate  established  at  MS  niA. 

7.  SINgle  Channel  Ground/Air  Radio  System  (SINCGARS);  this  program  had  no  EMD  phase 
and  therefore  is  not  comparable. 

8.  BLRIP  Report  written  and  included  in  1990  DOT&E  Annual  Report,  but  only  submitted  to 
Congress  in  February  1994. 

9.  Cost  data  from  Blue  Books;  “EMD  RDT&E  $”  are  totals,  not  RDT&E  only. 

10.  Joint  Tactical  Information  Display  System;  OTA  performance  rating  based  on  OPEVAL  Re¬ 
port  (OT-IIE)  dated  19  October  1994. 

1 1 .  Full  Rate  Production  ADM  issued  1 8  October  1995  without  a  formal  DAB  meeting. 

12.  Not  used. 

13.  Not  used. 

14.  Forward  Area  Air  Defense  Command,  Control,  Communication,  Intelligence  (FAAD  C^I)  and 
Ground  Based  Sensor  (GBS). 

15.  Advanced  Field  Artillery  Tactical  Data  System  (AFATDS). 

16.  OPEVAL  results  are  from  FOT&E  dated  28  Feb  96. 

17.  Enhanced  Position  Location  and  Reporting  System  (EPLRS). 

18.  Joint  Tactical  Information  Display  System  (JTIDS)  Class  2  Terminals. 

19.  Airborne  Warning  and  Control  System-Radar  System  Improvement  Program  (AWACS-RSIP). 

20.  Joint  Direct  Attack  Munition  (JDAM). 

2 1 .  Division  Air  Defense  (DIVAD) ;  this  program  was  terminated  on  27  Aug  85  after  64  units  were 
delivered  and  3  years  after  MS  III. 

22.  lOT&E  results  are  from  System  Assessment  dated  Nov  97. 

23.  Program  didn’t  require  a  SAR;  therefore,  no  cost,  schedule  or  extended  data  base  information 
is  available. 

24.  Final  SAR  data  currently  not  available. 

25.  This  program  was  91%  through  EMD  before  its  first  SAR  was  issued.  No  complete  EMD 
trend  data  is  available. 

26.  Not  used. 

27.  Not  used. 

28.  lOT&E  data  not  available  from  SAR  schedule. 

29.  This  program  had  multiple  cost  and  technical  variations  and  aspects.  It  is  difficult  to  analyze 
using  SAR  data  only. 

30.  Combat  Service  Support  Control  System  (CSSCS). 
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3 1 .  Ship  Self-Defense  System  (SSDS). 

32.  Joint  Stand-Off  Weapon  (JSOW). 

33.  Secure,  Mobile,  Anti-Jam,  Reliable  Tactical  Terminal  (SMART-T). 

34.  Close  Combat  Tactical  Trainer  (CCTT). 

35.  Army  Tactical  Missile  System  (ATACMS). 

36.  B-IB  Block  D  Conventional  Munitions  Upgrade  Program  (CMUP). 

37.  Not  used.  ...  ,  . 

38.  This  program  was  terminated  while  in  EMD.  The  data  contained  within  the  SARs  make  it 

difficult  to  determine  cost  and  schedule  overruns  accurately.  Analyst  estimates  are  used  and 
explained  in  the  research  office  files.  Cost  and  schedule  success  ratings,  however,  are  accurate. 
Only  RDT&E  funds  are  shown  in  the  SARs. 

39.  Costs  are  then-year  RDT&E  for  a/c  and  Defensive  Avionics  System  CORE  program  until 
terminated.  No  MS  IH  stated  in  S  AR;  end  of  lOT&E  used  for  schedule  overrun. 

40.  MS  niB  D AB  held  May  9 1  resulted  in  approval  to  continue  LRIP  until  the  BLRIP  report  was 
submitted  to  Congress.  This  date  is  used  to  end  EMD  for  this  program. 

41.  This  was  an  NDI  program  and  the  use  of  RDT&E  funds  for  cost  considerations  may  no  be 

representative. 

42.  The  production  program  was  terminated.  End  of  EMD  is  taken  as  the  date  the  contract  for 
remaining  LRIP  items  was  signed. 

43.  MS  n  date  was  Apr  83,  but  the  first  SAR  was  12/89  due  to  special  access  program  start. 

44.  Program  featured  NDI  strategy  and  the  SARs  went  from  a  planning  estimate  to  the  production 
estimate  without  a  development  estimate. 

45.  MS  II  was  Sept  82,  but  the  first  SAR  was  12/85. 

46.  MS  n  was  Feb  80,  but  the  first  SAR  was  12/83. 
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Program 


ALCM _ 

TTC-39  (TRI^TAC  Switch) _ 

iGTYORKgMiDI 

JTIDS  (Class  2  terminals 


AV-8B  (Harrier  II) 
OH-58D/AHIP 
Maverick  AGM-65D  (HR) 
LANTIRN  (Nav&Tet) 


Trident  II  msl  (D-5) _ 

Tomahawk  (TASM  &  TLAM-C) 
CV  HELD  (SH-60F) 

ASAT(AF) _ 

..MK.48ADCAP . 

Avenger  (Ped  Mtd  Stinger) 

.^CMS . . . 

ACM _ 

MK  50  Torpedo  (ALWT) 

Navstar  GPS/U.E. _ 

AMRAAM 
PLS  (FHTV)  (NDI) 

FAADS  LOS-F~H  (ADATS) 
B-IB  Lancer  (ALQ-161) 

ASPJ  (ALQ-165) 

. 


VLASROC _ 

NESP  (ANA/SC-38V) _ 

Paladin  M109A6 
SINCGARS 

Rolling  Airframe  Missile _ 

MlA2Tank 

'T45fs . 3 . 

JTIDS  (C1272H)  . 

FAAD  C3I  and  GBS 

FMTV _ 

Longbow  Apache  (AH- 1) 

C-17A 

AFATDS 

Mk  48  ADCAPS  Mods _ 

SFW _ 

JST^RS 


EPLRS _ 

csscs . . 

Javelin 

^  . 

SSDS  . . . . 

JSOW 

SMART-T 

CCTT 

JDAM . 1  ’’’’ 

E-2C  Update 

. 

B-lBBlkDCMUP 
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Program  System 
Number  Type 


Mis/Mu-A 

Elec-CNR-G 

...._.MisMu.:.G 

Elec-CNR-A 


^Aircraft 

Elec-CNR-A 

_ Mis/Mu- A 

Elec-CNR-A 


Mis/Mu-G 

Mis/Mu-G 

ilec:CNR^^^^^ 

Mis/Mu-A 


Mis/Mu-G 

Mis/Mu-G 

Mis/Mu-G 

Mis/Mu-A 


Mis/Mu-G 
Elec-CNR-A 
Mis/Mu-A 
Vehicle 
Mis/Mu-G 
Elec-EW-A 
Elec-EW-A 
. AIS . 


Mis/Mu-G 

Elec-CNR-G 

Vehicle 

"ll^^CNR^^^^ 

Mis/Mu-G 

y^icle 

Aircraft 

ei^-cnr-a"' 

’"ei^-cnr-g' 

Vehicle 

Elec-CNR-A 

Aircraft 

Elec-CNR-G 

Mis/Mu-G 

Mis/Mu-A 

Mis/Mu-G 

Elec-CI^A 


Elec-CNR-G 

...Elec:CNR:G 

Mis/Mu-G 

'ilec-CNRX 

Eiec-CNR-G 

Mis/Mu-A 

Elec-CNR-G 

Elec-CNR-G 

MisM 

Aircraft 

Mis/Mu-G 

Elec-CNR-A 
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Notes  I  Service 


8  I  9 


Cost  Cost  Schedule 
Success  Percent  Success 
Overrun 


45 

71 . 

64 

319 


27 

13 

7 

-5 


Apr-95 

Aug-95 

. .Oct:95 

Nov-95 

Dec-95 

Mar-96 

Jun-96 

Jul-96 

SeD-96 


Mar-97 

j^r:97 . 

May-97 

'Sep-97~ 

Mar-98 

Oct-98 

Nov-98 

Dec-98 


17,  23 

Army 

23,  30 

Army 

22 

Army 

19,  24 

AF 

23,24,31 

24,  32 

AF 

24,  33 

Army 

24,  34 

Army 

20,  24 

9 

Navy 

24,  35 

Army 

24,  36 

C-7 


Program 


. 

SGT  YORk  gun  (DI\^) 
JTIDS  (Class  2  terminals) 

AV-SB  (Harrier  II) . . . 

OH-58D/AH1P . . . . . . 

Maverick  AGM~65D  (HR) 
. . 

Trident  II  msl  (D-5) _ 


10 


Schedule 

Percent 

Overrun 


. 28 _ 

J2_ . 

. ~'59 

136 

1 . . 

. 11 . 

175 

__34 . 

2 


Tomahawk  (TASM  &  TLAM-C) . I . „_.167 

_  I  ^ 

ASAT(AF)  _ 

MK48  ADCAP _ _ _ _ _ 

Avenger  (PedMtdStinjge^ _ 

i^ACMS  _ 

VCM  . . . . . . . . 

MK  50  Torpedo  (ALWT) _ 

Navstar  GPS/ILE. _ _ _ 

AMRAAM _ 

PLS  (FHTV)  (NDI)  . . . . 

i^ADS  LOS-F-H  (ADAT  S) . 

B-IB Lancer  (ALQ-161)  _ 

ASPj  (ALO-T65) _ 

CHCS  S/W  Ver  4.01 _ 

VLASROC  Z 

PaiadTn  M169A6 _ 

SINCGARS _ 

^Uing  Airtane  Mi^le_ 

Ml  A2  Tank . . . . . . . . 

Hisfs  _ _ _ 

JTIDS  (Cl  2/2H) _ 

FAAD  C3I  andGBS . 

FMTV'  _ 

Longbow_Apach£(AH^^ _ 

C-17A  _ _ _ _ 

AFATOS2 _ 

Mk  48  ADCAPS  Mods 

SFW  _ _ _ _ _ _ _ 

Std  MsHS;M-2)  Bik  III _ _ _ 

JSTARS _ 

EPLRS _ _ _ _ _ _ _ 

CSSCS _ _ _ _ _ _ _ 

Javelin  . . . . . . . . . 

AWACS-RSIP _ 

SSDS  . . . . . . . . 

JSOW . . . . . . . _ . . . 

SMART-T . . . . .  . 

C'CTT 

JDAM  _ _ _ _ 

E-2C  Update _ 

ATACMS  Blk  I 

B-1 B  Blk  5  CMUP 
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Program 


ALCM _ _ 

SGTYQRK  gun  (DI\^) _ 

JTIDS  (Class  2  terminals) 


AV-8B  (Harrier  II) _ 

QH-58D/AHIP _ 

Maverick  AGM-65D  (HR) _ 

&  Tgt) _ 

Trident  II  msl  (D-5) _ 

Tomahawk  (TASM  &  TLAM-C) 


CV  HELP  (SH-60F) . . . 


MK  48  ADCAP _ 

Avenger  (Ped  Mtd  Stinger) 

ATACMS 

ACM 

MK  50  Torpedo  (ALWT) 
Navstar  GPS/U.E. 
AMRAAM 

. . 

B-IB  Lancer  (ALQ-161) 
ASPJ  (ALQ-165) 

CHCSS^  . . 


VLASROC _ 

NESP  (ANA^SC-38V) 
Paladin  Mip9A6 
SINCGARS 

Rolling  Airframe  Missile 

MlA2Tank _ 

T45TS 

JTIDS  (Cl  2/2H) 
MAD^^ 

FMTV _ 

. 

AFATDS 

Mk  48  ADCAPS  Mods 

SFW _ 

StdMsL{SM:2)B^^ 

JSTARS _ 

EPLRS _ 

CSSCS _ 

Javelin . . 

. . 

SSDS . . 

JfSOW . . 

SMAR^ . 

CCTJ  . . . 

JDAM 
E-2C  Update 

ATACMS  Blk  I _ 

B-IB  BlkDCMUP 


X 

Jan-82 

X 

. X . 

. "x . 

Dec-88 

Sep-84 


X 

Dec-92 

X 

. X . 

. X . 

Apr-94 

Oct-90 


Noy-90 

Sep-9i 

Jan-92 

. XipjJ^2 

Dec-92 

Dec-92 

pec-92 

Dec-92 

Dec-92 


May-93 

May-93 

Jun-93 

Ilep:93 

Dec-93 

Apr-94 

Jan-95 


18 


EMD 

Duration 

Actual/Plan 


1.28 

1.42 

. . 1,59 . . 

^35 


1.01 

1.11 

. 2,75 . 

1.34 

1.02 

2.67 . 

1.66 

1.23 


1.48 

1-18 

1^33 

1.15 

1.46 

1.30 

. ] . 2^27 . 

. . L58 . 

. “l^8 . 

2.30 
2.70 


19 


EMD 

Overrun 

Months 


Jul-86 

May-88 

Pec-90 

]feb:85l 

Sep-89 

Sep-82 

Nov-85 

Jun-89 

Apr-88 


Jun-89 
Dec- 88 


Jan-92 

Jan-91 

OcL96 

”i^9r 

Apr-94 

Jul-88 

Sep-94 

Sep:93 

Dec-91 


Jun-94 

. X . 


Apr-99 

Nov-99 

May-01 

Aue-01 


Apr-95 

Aug-95 

pct:95 

. Npv:95 . 

pec-95 

Mar-96 

Jun-96 

Jul:96 

Sep-96 

Mar-97 

Apr-97 

MaY-97 

Sep-97 

Mar-98 

pct-98 

....Npy-_98 

Dec-98 
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Program 


..M.CM . - . . . - . 

TTC-39  (TRI-TACiLwkcK _ _ 

SGT  YORK  gun  (DIVAD) 


AV-8B  (Harrier  II) _ 

OH-58D/AHIP  . . . . . . 

Maverick  AGM-65D  (HR) 


MNmNiNayA.Tgfi...„ . 

Trident  II..m  . . . . 

j^nmahawk  (TASM  &  TLAM-C) 

CV  HELP  rSH-60F) _ 

ASAT  f  AF) _ 


..MK48ADCAP . . . . . 

..Avenger..(Ped^^M  . 

ATACMS _ 

ACM  .  . . . . 

MK  50  Torpedo  (ALWT) 
Navstar  GPS/U.E. 

AMRAAM _ _ 

PLS  (FHTV)  (NDI 


FAADS  LOS-F-H  (ADATS) 
B-IB  Lancer  (ALO-161)  . 

ASPJ(Ad5  165^ . . . 

CHCS  S/W  Ver  4.01 _ _ 


VLASROC _ _ 

. . . 

Paladin  M109A6 . . . 

SINCGAR^ _ _ 

Rolling  Airframe  Missile _ 

MlA2Tank  _ _ _ 

T45TS . . 

JTIDS  (Cl  272H) _ 

FAAD  GBS 

mfw . . 

Longbow  Apache  (AH-;!) . . 

C:17A . . . . 

AFATDS _ 

Mk  48  ADCAPS  Mods _ 


SPW . . . . . 

Std  Msl  (SM-2)  Blk  in _ 

JSTARS _ 

EPLRS _ _ 

CSSCS _ _ _ 

..Javelin . . . . . 

AWACS-RSIP _ 

...SS.D.S.  . . . . 

JSQW  „  „„ 

SMART-T_  _  . .  „„ 

CCTT 

JDAM 

E-2C  Update _ 

,^CMS.BlkI . . . . . 

B-IB  BlkDCMUP 


21 


EMD 
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